“Knock! Knock! Who is it? ” Probabilistic Person Identification in TV-Series

We describe a probabilistic method for identifying char-acters in TV series or movies. We aim at labeling every character appearance, and not only those where a face can be detected. Consequently, our basic unit of appearance is a person track (as opposed to a face track). We model each TV series episode as a Markov Random Field, integrating face recognition, clothing appearance, speaker recognition and contextual constraints in a probabilistic manner. The iden-tification task is then formulated as an energy minimization problem. In order to identify tracks without faces, we learn clothing models by adapting available face recognition re-sults. Within a scene, as indicated by prior analysis of the temporal structure of the TV series, clothing features are combined by agglomerative clustering. We evaluate our ap-proach on the first 6 episodes of The Big Bang Theory and achieve an absolute improvement of 20 % for person identi-fication and 12 % for face recognition. 1